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Foreword 


E-learning in the European Higher Education Area has stampeded its way to the 
foreground of the Quality Assurance (QA) forum, and has become a key issue among 
quality assurance agencies and institutions in the European Higher Education Area 
(EHEA). Because internet-based learning is currently such a relevant topic, there is 
a dire need for the creation of a common language and guidelines amongst all QA 
agencies in order to proceed in a collectively positive direction in regards to developing 
a quality culture within the frame of E-learning further. 

This report gives a general overview of the matters discussed and challenges faced 
within the sector of quality assurance in E-learning. The workshop ENQA held in 
coordination with the National Agency for Higher Education, in Sigtuna, Sweden in 
October, 2009 created a dialogue between institutions, quality assurance agencies, 
students and other stakeholders who are directly affected by the quality of E-learning, 
and the need for continuous improvement in this field that will foster positive 
outcomes. 

Achim Hopbach, 

President 

European Association for Quality Assurance in Higher Education (ENQA) 


Introduction 


Josep Grifoll, AQU Catalunya, Spain, and Michele Soinila, ENQA Secretariat 

Over the past decade as technology coupled with the increasingly frequent use of the 
Internet becomes the forefront of business and academia, E-learning has emerged onto 
the global higher education stage as a leading means of gaining a respected education 
in the European Higher Education Area (EHEA). The question that remains is how do 
QA agencies monitor existing E-learning provision and develop future provision in a 
reliable and efficient manner? 

The European Standards and Guidelines for Quality Assurance in the European 
Higher Education Area (ESG) have laid the foundation for web-based learning 
provisions and regulations. With the appropriate interpretation, quality assurance 
agencies could use the ESG as a backbone document and create additional material that 
would aide quality assurance agencies in monitoring the progress and development of 
E-learning. 

This workshop provided a platform to discuss the underlying issues and internal 
challenges related to internet based learning, including the need for a “common 
language” and an integrated approach to E-learning - in which quality assurance 
agencies could collectively reference when creating provision related to E-learning. 
Many other topics were presented and touched upon, some of them are as follows: 
experiences in internal quality assurance in E-learning, web-based programmes offered 
at different institutions, implementation of efficient web-based tools, and the need for 
specific international accreditation and evaluation in E-learning. 


The following report includes five articles, submitted by the speakers and based on 
their presentations, which introduces the main topics for discussion and proposals for 
improved monitoring of Quality within the E-learning realm. Additionally, this report 
provides interesting perspectives from different institutions that have implemented 
E-learning provision, and discovered ways in which to effectively monitor web-based 
activity. 


Chapter 1: 

E-learning in the context of the 
Standards and Guidelines for Quality 
Assurance in the European Higher 
Education Area (ESG) 

Josep Grifoll, AQU Catalunya, Spain 


1.1 Introduction 

In my presentation, I suggest that the use of Standards and Guidelines for Quality 
Assurance in the European Higher Education Area (ESG1) is not contradictory to the 
generation of relevant opportunities for innovation and enhancement of the quality 
assurance process in Higher Education, and in e-learning in particular. In fact, the ESG 
can be seen as a catalyser for the definition of new concepts of quality in the coming 
future. 

1.2 E-learning and the ESG 

We can start by formulating the following question: are programmes based on 
e-learning methods requiring a reformulation of the ESG? If we understand the ESG 
as a frame for quality assurance, the answer to that question is, in my opinion, no. 

Let’s say “no” for the moment. Reasonably, what will be needed, sooner or later within 
the e-learning realm, is a general reformulation of some current educational policies 
and practices. This happens in a world in which the information and communication 
technologies are making the costs of getting facts, data and, presumably, skills 
dramatically lower. The vast amount of information available is giving people far more 
opportunities to boost their knowledge, not only through enrolment in a programme 
as a student, but also through alternative paths. In the same way that we assume that 
strategies for training people are expected to be reformulated drastically in the future, 
the standards will likely be subject to adaptation as well. 

My view is that the ESG are well defined. I suggest reading and using the standards 
to tackle the issue of quality assurance of e-learning programmes, keeping two ideas in 
mind: 

a. E-learning should not be an exclusive methodology for particular programmes. 

On the contrary, e-learning strategies are more and more essential in modern 
educational activities. 

We should also keep in mind that quality assurance should not forget the way 
in which information and communication technologies are creating alternative 
opportunities for both teaching and learning in universities, with the associated 
challenges for our higher education sector. 


Thus, quality assurance policies need to formulate questions on how far e-learning 
methods are included in all study programmes, and on the adequacy between new 
technologies and the emerging new educational approaches, taking into consideration 
concepts such as efficiency in teaching, effectiveness in learning or equity in education. 

At the moment, it is quite difficult to predict the evolution of new technologies that 
can be used in teaching and learning. They are changing so fast that even specialists 
find it difficult to make a general composition of what can be expected in the next 20 
years. This introduces a big disadvantage to the planning task of our policymakers and 
university managers. 

Take, for example, the evolution in the capacity of computers to work with data, or 
the innovations in artificial intelligence. How can these tools provoke changes in the 
way we process information, in the way we teach, in the way we learn, or even in the 
way we organise research activities? 

As another example, let’s think about the developments in translation technology. 
They could produce huge consequences, not only in education but also in research, and, 
undoubtedly, in the way our societies are organised. Gaining freedom and reducing 
communication barriers between individuals is not the only outcome, but also giving 
new opportunities for the smallest cultures to gain international visibility. Can we 
imagine students following study programmes delivered in multiple languages and 
translated automatically into their mother tongues? It doesn’t exist today, but it could 
certainly happen in the future. Those technologies will undoubtedly make it easier to 
set up new and diversified learning communities, working with new parameters that 
can affect the way we perceive quality in teaching and learning. 

Or, let’s take the developments on virtual reality. Video game technology is 
fascinating, and it can be used for providing new educational applications, for example, 
in creating virtual laboratories for our study programmes. 

Electronic ink is an expected and desired technology not only for reading purposes, 
but also for creating portable libraries for everybody. Have we agreed about the most 
relevant indicators of what is a good university library in that context? 

Social networks based on Internet technology are already used in e-learning 
programmes demonstrating fruitful results. What is quality in those networks? 

Again, we need to think of introducing new concepts of quality. For example, those 
networks seem to become the future campus of our universities, and taking care 
of the « electronic arena » of those campuses can reward a lot of advantages at an 
institutional and individual level. If we wish to protect and promote the value of 
freedom, for instance, in our educational communities, we need to be aware of the 
potential vulnerability of those networks. We need to protect personal data and to 
introduce stronger ethical requirements for students. 

‘Googling’ technologies, and the development of new, more advanced robots for 
searching and processing data and facts stored in computers, is another important 
point to be considered. Do we need new concepts to define learning and teaching? 
Traditional teaching scheme can be useful for freshmen, but perhaps more experienced 
students will be interested in universities that provide unique experiences not only to 
learn the current knowledge in a particular way, but also to contribute to the creation 
of new knowledge during their enrolment. Are future students expected to be rather 
explorers than learners? 


There will be new technologies for data transmission with new networks, with 
cheaper access and stronger capacity. Can you imagine a future in which people can 
transmit data with no limits? In that world, universities should develop adequate 
infrastructures to become nexuses, or what we could call the cyber agora, where people 
interested in scientific developments meet in order to keep their central position in the 
knowledge society. 

Or, what are the expected effects of using new interfaces between people and 
computers that make acquiring general knowledge easier? In that world, how will we 
value people’s contribution to the progress of society? 

1.3 E-learning in light of the basic principles of the ESG 

A second relevant reflection on the consistency of the ESG for e-learning can be done 
by observing their basic principles. This second consideration leads us towards some 
interesting points, in which the examination of those principles raises new questions 
for the future. 

Take, for example, those e-learning programmes that can be delivered online, and 
offered to those students who could be traditionally enrolled in distance education. 

The first basic principle declares that providers of higher education have the primary 
responsibility for the quality of their provision and its assurance. This is a principle 
that should be developed and implemented in a deeper way. However, e-learning 
programmes are progressively enrolling students and hiring teachers situated in 
different countries. Facing this situation, how do we match the primary responsibility 
with the needed “secondary” responsibility of QA agencies and other stakeholders? 

How will international e-learning programmes be externally assessed? 

The second basic principle of the ESG states that the interests of society in the 
quality and standards of higher education need to be safeguarded; the concept of 
society here, and taking into account again the possibilities of e-learning programmes 
to be delivered worldwide, needs also deep reflection. Who represents the society? 

That is important if we wish to include the voice of society in the quality of study 
programmes, and in the definition of new proposals. 

The definition of society is also relevant when we have discussions on funding 
schemes for education. Who is investing in education in our societies, and who is 
getting the benefits of that education? Of course students invest their time and money 
to study, but governments are also investing resources on behalf of their national 
taxpayers. How does e-learning fit in a world of national policies for public goods, with 
low barriers to work at an international level? 

Another interesting basic principle points out that the quality of academic 
programmes needs to be developed and improved for students and other beneficiaries 
of higher education across the EHEA. E-learning can offer interesting opportunities 
for students and other beneficiaries of Higher Education. See the potential benefits of 
the academic mobility in higher education. E-learning could be a very comfortable way 
to be enrolled in a foreign degree with other European students attending the same 
virtual classroom. 

Current technology can be used and developed to set up international joint 
programmes based on e-learning. This is something that could be very interesting, not 
only in Europe, but all over the world, providing a sort of virtual mobility that is much 
more affordable for students and public funds. 


Another expected outcome for e-learning programmes is better access to higher 
education for people with disabilities. See developments on text-to-speech technologies. 

A remark concerning the forth principle, referring to the need of efficient and 
effective organisational structures within academic programmes, can be provided 
and supported. E-learning programmes can be organised in a very different way, if 
we take advantage of the use of new technologies. As technologies used in e-learning 
are rapidly changing, quality assurance strategies need to pay attention to that fact. I 
personally think that e-learning offers a chance to improve the way departments and 
institutions are organised. There are limitations, of course, but the constitution of 
virtual communities for teaching and learning with international and diverse expertise 
is easier with new technologies. Thus, quality assurance should take into account 
the possible deficits in the use of new technologies for efficiency and effectiveness in 
organisations. 

Transparency and the use of external expertise in quality assurance processes are 
important; in all kinds of programmes, and in e-learning in particular, the question 
of transparency and the use of external expertise as a way to increase confidence on 
the quality are crucial. One should do the exercise of getting information on academic 
study programmes delivered in European universities, and he or she will come to 
the conclusion that there are enormous opportunities for improvement on that issue. 
Moreover, if a reader is interested in knowing who is externally granting the quality of 
those programmes, the situation is confusing. There is, therefore, a lot to do for our QA 
agencies. 

The seventh principle states that accountability processes should be developed, 
through which higher education institutions can demonstrate their accountability, 
including accountability for the investment of public and private money; as e-learning 
makes it easier to deliver programmes at the European level, accountability for public 
and private investors should be carefully treated and connected with the expectations 
of different national and international stakeholders. 

The ESG basic principles also state that institutions should be able to demonstrate 
their quality at the national and international level; this is a principle that should 
generally apply to all higher education, but specifically to e-learning programmes which 
are ready to admit students from different countries. 

1.4 In conclusion 

There are, however, other questions emerging from the ESG. We have seen how 
these standards and principles generate opportunities to reflect and to develop 
quality assurance methods with possible new parameters and indicators. Promoting 
opportunities for innovation in QA of e-learning is a priority. This is critical for 
knowledge acquisition in the future society in which the creation of new communities 
will set new demands for universities to provide not only better understanding of the 
reality, but also renewed chances and experiences to go far in the creation of new 
knowledge. 


Chapter 2 : 

How to assess an e-learning 
institution: Methodology, design and 
implementation 

Esther Huertas, Anna Prades and Sebastian Rodriguez AQU Catalunya, Spain 1 

Abstract 

There are certain features in the assessment of distance learning in higher education 
that need to be particularly taken into account. A methodology adapted for an 
e-learning institution (Catalan Open University, UOC) was designed by the Quality 
Assurance Agency of Catalunya (AQU Catalunya) with the institution’s characteristics 
(student profile, teaching methodology, teaching staff, etc.) incorporated into the 
evaluation model. In addition to a distinction in the assessment methodology between 
institutional evaluation and programme evaluation, the specific aspects of e-learning 
also called for adaptations of the evaluation process to be made. Training for the 
external review panel and on-line access to the university’s virtual campus were thus 
key aspects throughout the evaluation process. This paper looks at the methodology’s 
potentialities and limitations following the two years of its use and the assessment of 
nine different degree programmes. 

2.1. Introduction 

Quality assurance in higher education is fully complied with the conventional 
universities, whereas quality assessment processes in e-learning programmes are 
only now becoming more widespread in Europe. The national agencies in Norway 
(NOKUT) and Sweden (NAHE) have developed small-scale projects on quality criteria 
for e-learning, and in the UK (QAA), guidelines have been drawn up on the quality 
assessment of distance learning. Nevertheless, e-learning quality is not included as a 
regular or integral part of national quality reviews in any country, nor is any emphasis 
placed on the standards and guidelines established by the European Association for 
Quality Assurance in Higher Education (ENQA) on the quality in e-learning. Other 
organisations, such as the National Association for Developmental Education (NADE, 
Norway), the Joint Information Systems Committee (JISC, UK), and the Higher 
Education Academy (HEA, UK) have focused on the methodological development of 
e-learning assessment (publication by Swedish National Agency for Higher Education, 
2008 ). 

Other experiences include the Australasian Council on Open, Distance and 
E-Learning (ACODE), which has published extensive benchmarks with the aim of 
influencing policy and practice at the institutional, national and international level. 

In the U.S., the Council for Higher Education Accreditation (CHEA) has drawn up 


1 AQU Catalunya is grateful to the Catalan Open University, and in particular to Maria Taulats and Isabel Sola, for all their 
suggestions during the process regarding the enhancement of the assessment design of distance learning programmes. 


guidelines for the accreditation and assurance of quality in distance learning, and 
the Distance Education and Training Council (DETC) has defined, maintained and 
promoted education excellence in distance education. 

There has been practically no experience with the assessment of e-learning quality 
in Europe. The Open University (UK) has been assessed, but according to the same 
national quality criteria as other British institutions of higher education. In Catalonia 
(Spain), an e-learning evaluation project is already finished (AQU Catalunya) at the 
higher education level, the main objective of which is the formative assessment of the 
Catalan Open University (UOC). 

This paper presents the methodological design implemented in the assessment of 
a fully virtual university, focusing on adaptation of the methodology, together with a 
discussion on the main potentialities and limitations of the process. 

2.2. Design of the methodological model 

2.2.1. CHARACTERISTICS OF E-LEARNING 

The evaluation methodology designed by AQU Catalunya follows a system based 
on the European model adapted to the evaluation culture within the university and 
social context of Catalonia. Before setting up the methodology design applied to an 
e-learning institution, it was necessary to study the differences between e-learning and 
conventional higher education, and more particularly, the specific characteristics of the 
university being assessed (in this case, the UOC). 

Various challenges exist when assessing a virtual university. Bearing in mind the 
structure of the CIPP (Context, Inputs, Process and Product Evaluations) model for 
evaluation (Stufflebeam, 2003), the main aspects that can be pointed out are as follows: 

a. Context 

Context evaluation is different from that of a conventional university, and it 
emphasises the specific characteristics of e-learning as to conventional higher 
education. On-line distance study has potentialities, on the one hand, but it also 
suffers from limitations, such as the type of degree that can be obtained. 

In terms of organisation, the Catalan Open University is a private university 
that has a structure which is more like that of a company than the management 
and administration of a conventional university. Together with the fact that it has 
a small body of teaching staff, this makes it a highly centralised institution with 
regard to information and quality assurance mechanisms. 

b. Inputs 

The profile of students enrolled in an e-learning education is different from 
that of students attending a bricks-and-mortar university. E-learning students 
usually work full time (they are employed in the labour market), they have family 
responsibilities and they tend to be more mature. The teaching staff profile is 
also different from that of conventional universities, with the UOC having its 
own “resident” teaching staff (director of studies, programme director and course 
coordinator), as well as collaborating teaching staff (student counsellors and 
tutors). The UOC’s own teaching staff propose courses, define the contents and 
aims, look for authors for the teaching materials, select and coordinate student 
counsellors, etc. Collaborating teaching staff consists of two posts: the student 


counsellor and the tutor. The student counsellor gives incentive and impetus to 
learning activities from the very beginning through assessment (by proposing and 
monitoring the student’s activity, moderating discussions and debates, resolving 
doubts regarding the subject, etc.). The tutor supports and advises the students on 
matters connected with the running of the virtual campus and course enrolment, 
and gives guidance regarding possible professional opportunities. 

Lastly, technology infrastructure forms the core of a virtual university, as the 
university has to guarantee that the services for study and learning purposes are 
satisfactory. 

c. Process 

The main difference concerning the delivery process is the high degree of 
homogeneity. All classrooms used for the same subject have exactly the same 
learning documentation, tools (forum, guidance, etc.) and assessment process. 

Distance learning education implies a high level of teaching process 
homogeneity: the same author for all materials in one particular subject, the 
same learning and assessment activities, the same student support system for 
all programmes, etc. This degree of homogenisation has advantages, such as the 
fact that the institution can make a cascade of changes quickly and effectively, 
although it also implies risk, such as the hegemony of a single culture to the 
detriment of plurality, as well as the possible devaluation of teachers as mere 
mediators of knowledge described by UNESCO (1998). 

d. Product evaluation 

Product evaluation identifies and assesses three kinds of outcomes: academic 
outcomes (progress rates, drop-outs, etc.), personal outcomes (skill development) 
and professional outcomes (employment rates and adequacy, etc.). It is important 
to state that the evaluation of e-learning programmes should be of the same 
quality as that of non-distance learning degrees (i.e. conventional degree 
programmes). 

2.2.2. METHODOLOGY DEVELOPMENT 

The aims of quality assessment of degree programmes are to promote quality, and to 
provide valid and objective information on university services to the society (in other 
words, accountability). 

In the case at hand, methodology development started with the adaptation of the 
design of distance learning education and the structure of the Open University. The 
main aspects considered in the new model refer to the dividing of the assessment 
process into two units (an institutional level and a degree programme level); the 
establishing of dimensions or sections for each assessment level; and the setting of 
indicators, standards and evidence for each dimension. 

Given the nature of the UOC as an institution (with high degree of homogeneity in 
the educational process and the institution’s highly centralised nature), it was necessary 
to divide up the assessment of the institutional evaluation level, on the one hand, and 
of the degree programme evaluation level, on the other (see Figure 1). The institutional 
level is centralised, and includes all aspects that are common to all degree programmes 


(mission, vision, delivery system, and infrastructure), with special emphasis on quality 
assurance policies and mechanisms, including the information systems that support 
these mechanisms. The other level, which is specific to each degree programme, 
specifies how the aspects, policies and general mechanisms work (the strategic position 
of the degree, study programme, instruction design, disciplinary (fields of knowledge) 
dimension, and outcomes). The relationship between the institutional evaluation and 
the degree programme evaluation is given in Table 1. 
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Figure 1. Evaluation units designed within the framework of the assessment of a fully virtual 
university (AQU, 2007a; AQU, 2007b) 


Table 1. Relationship between the dimensions (institution/degree programme) within the 
framework of evaluation and distance learning 


DIMENSION 


INSTITUTION DEGREE PROGRAMME 


1. INSTITUTIONAL MISSION AND VISION 


1.1. Institutional mission 

• Appropriateness of the programme and student 
profile to the institutional mission and vision 
(Section on the strategic position of the degree). 




2. QUALITY ASSURANCE MECHANISMS 


2.1. Institutional mission and vision 

• N/A 

2.2. System capacity: 

adequacy of policies regarding students, teaching 
staff, infrastructure and external relations 

• Adequacy of students, teaching and professional 
staff profile (section on the strategic position of 
the degree). 

• Appropriateness of technical set-up for instruction 
section (instruction design section). 

• Comparison of the proposed programme with 
others, e.g. number and quality of students, type 
of teaching staff, etc. (section on the strategic 
position of the degree). 

• Adequacy of interpersonal communication 
systems (instruction design section) 

2.3. Internal and external strategic position: 
adequacy of the information-gathering mechanisms, 
as well as the analysis and decision-taking circuit 

• Adequacy of internal and external strategic 
position (section on the strategic position of the 
degree). 

2.4. Learning outcomes and study programme: 
adequacy of planning mechanisms and learning 
outcomes’ quality assurance mechanisms 

• Adequacy of the definition of learning outcomes 
(study programme section). 

• Adequacy of the study programme (study 
programme section). 

2.5. Instruction design: 

adequacy of planning mechanisms and programme 
specifications’ quality assurance mechanisms 

• Adequacy of the activities (instruction design 
section). 

• Adequacy of the degree organisation (instruction 
design section). 

• Adequacy of student orientation, tutoring and 
advisory system (instruction design section). 

2.6. Learning assessment: 

adequacy of planning and quality assurance 

mechanism regarding student assessment 

• Adequacy of the assessment system (learning 
assessment section). 

2.7. Outcomes: 

academic, professional and personal 

• Adequacy of academic, professional and personal 
outcomes section. 


N/A: Not applicable 

Indicators, standards and evidence are based on those specified for on-line education 
by the Council for Higher Education Accreditation (CHEA), the Institute for 
Higher Education Policy (IHEP) and the Western Cooperative for Educational 
Telecommunications (WCET). Account was also taken of the accreditation standards 
for the degree programmes participating in the European Higher Education Area 
(EHEA) pilot programme implemented by AQU Catalunya, and the standards for 
internal quality assurance in the EHEA adopted under the Bergen Declaration (2005). 


2.2.3. EVALUATION PROCESS 

The self-evaluation committee analyses all the dimensions described in the 
methodology guide, and draws up its self-evaluation report. The external panel reviews 
the completed report prior to the site visit to the institution where the external review 
panel interviews the various stakeholders. This process is supplemented with on-line 
interviews, thereby enabling all stakeholder groups to take part in the assessment 
process. The external experts receive training on the virtual campus prior to the site 
visit in order to be able to make best use of the on-line interviews. The process is shown 
in Figure 2. 



Figure 2. Process design for the evaluation of e-learning institutions and degree programmes. 

2.3. Potentialities and limitations of the process 

There are certain benefits in dividing the evaluation into two units (institutional 
evaluation and degree programme evaluation), as compared to a conventional 
assessment. The external report on the institutional evaluation (now complete) was 
used as an input for the degree programme evaluation, enabling the experts on the 
external panel to focus on the key aspects related to the degree programme evaluation. 

The process shows the importance of the external review panel being trained in 
an e-learning education system, as well as the specific nature of the university that 
is being assessed. The UOC educational model, the classification of teaching staff 
(UOC’s own teaching staff and collaborating student counsellors and tutors) and the 
importance of research are the foremost characteristics that, in this case, are different 
from conventional universities. Due to these differences, members of the external panel 
had access to all resources available to the students (e.g. library), and some classroom 
examples were included in order to show the organisation of forums and debates, 








counsellors’ comments, student assessment, etc. The availability of this resource and 
the fact that it is compulsory to use the university’s virtual campus are aspects that the 
experts assess very highly. 

The implementation of the process shows that on-line interviews are highly 
important for the external review panel. These interviews allow additional information 
not included in the self-evaluation document to be collated. It is worth mentioning that 
this stage in the process involves a lot of work by the experts, given the large number 
of interactions, especially during the student interviews. The external panel also works 
with the virtual campus and they can thereby check the soundness of the system. 

One significant limitation of e-learning education assessment is the lack of 
benchmarks within the context of evaluation. The UOC is a fully virtual university and 
there is no other institution with a similar structure in Spain. Moreover, information 
on other renowned e-learning universities is not readily available. 

Another minor weak point found during the institutional evaluation was the need to 
include economic information on the organisation. This information, which needs to be 
presented in the main sections, provides an understanding of the institution’s current 
situation. In addition, an assessment of the strategic plan should complete the mission 
and vision evaluation in order for a broader assessment to be made of the university’s 
situation in the near future. 
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Chapter 3: 

Modern E-learning: Qualitative 
education accessibility concept 

Yuri Rubin, AKKORK, Russia 


3.1. Quality and competitiveness 

Quality in education is one of the main issues examined by modern scholars and 
practitioners who operate on the international education and resources market. 

High quality is the main competitiveness indicator. In the fierce competitive 
environment, the management and staff of universities should efficiently manage the 
learning process, and take steps to improve their institutions’ competitiveness level, 
all of which is impossible if no steps are taken to enhance quality in education 2 . This 
is the reason why accreditation agencies involved in quality assurance and in quality 
evaluation in the field of education were rapidly expanding their presence on the 
market in the majority of the market-oriented economies during the last few decades. 

The quality of education reflects the relationship between learning (seen as a result, 
a process or as an education system) and the demands, goals, standards (regulations) 
and requirements set by individuals, businesses, organisations, local community 
members and the state at large. If we use the above approach, the term ‘quality of 
education’ should be broken up into the following terms that require a separate 
definition each: 

• Quality of teaching (learning process design, teaching methodology); 

• Quality of academic staff; 

• Quality of study programmes ; 

• Quality of equipment, maintenance and support rendered; quality characteristics 
of the learning environment; 

• Characteristics of students, school students, university entrants; 

• Quality of university management; 

• Quality of research. 

3.2. E-access and l-access to quality 

Meanwhile, modern E-learning is one of the typical results of the competition between 
universities. Its embedding as an element of the structured system makes it necessary 
to prevent extremes in approaches to quality evaluation and quality assurance: 
from the recognition of E-learning as the only mainstream of modern education to 
the unconditional allocation of E-learning to the educational underground, or its 
interpretation as a purely technological environment in which traditional content is 
realised. Today’s reality is integrated learning (I-learning) that combines the elements 
of traditional face-to-face learning and E-learning: integrated contents, the ways of 
content presentation, communication between students and professors, study methods, 

2 For more information, please see: Rubin Y. (2006) KoHKypemiMfl: ynopaflOHeHHoe B3anMOfleMCTBne b npocjDeccMOHajibHOM 

6M3Hece [Competition: structured interrelation in professional business], Moscow: Market DS; Rubin Y. (2008) Kypc 
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means of organisation and management of the educational process in universities. 
Therefore, the importance of the Sigtuna workshop theme is that quality assurance 
agencies have to be more sensitive to innovations associated with E-learning, and 
to find “E-learning segments” during the evaluation process of the educational 
programmes and universities. 

Modern I-learning could get appropriate recognition in the framework of the 
qualitative education accessibility concept. 

Integrated learning, where face-to-face and correspondence learning characteristics 
are integrated, finely combines all the best practices elaborated in the field of learning 
content, technology, HR management, university administration and management, 
including the best practices shown in E-learning, M-learning, on-line and off-line 
courses. 

Consistent introduction of E-learning into the learning process fosters the modern 
ICT introduction process, and creates a good environment for the integration 
of learning content, learning technology, various learning process designs and 
professional competence. That is why modern E-learning should be considered as a 
precursor and a part included into the integrated learning. 

I-learning elements have always been present in the learning modes used in Russia 
(both in the face-to-face and in the distant mode, and, of course, in the mode where 
these two are combined). Due to E-learning, I-learning now has a new function that 
enables resource integration and participants’ interaction in the learning process. Here 
the distant mode potential is integrated into the face-to-face mode potential. By using 
the distant mode, the remotely located learners make use of the principle of having an 
easy access to education. As a result, the overall quality of provision is improved. 

Such integration opens new horizons for any national education system in terms 
of improving and developing the learning content and technology, and using the 
professional competence of teaching staff in combination with high technologies, 
inter-university and international cooperation to ensure a good mobility of content, 
technology, teaching personnel and various learning modes. As Tim Priestman 
reasonably says, integrated learning ensures that the knowledge will be transferred to 
any point in any part of the world 3 . 

As a product of modern telecommunication technology-based systems, E-learning 
turns out to be an efficient tool for bridging the distance gap on the Internet. In fact, 
E-learning is not a remote learning tool; it is a tool for overcoming the distance gap 
as such. That is why distance learning and E-learning are not to be included into 
one category. The distance gap is completely bridged for the parties involved in an 
E-learning session within the framework of instructors-to-students and students- 
to-students interactions. If a rationalised approach to E-learning is used, reasonable 
managers of ‘state-of-the-art’ universities do not use a traditional break-up scheme in 
their descriptions of learning modes (face-to-face and correspondence courses). They 
speak about various forms of integrated learning. This mode includes the face-to-face 
mode used in combination with modern ICT, and this is what transfers the learning 
process into the virtual reality. 

When integrated learning is examined, E-learning is seen as a system that can be 
used by traditional universities, not as a tool to be used exclusively by those universities 

3 Tim Priestman (2007), Blended Learning in International MBA Programme: Partner Initiatives and Future Challenger in Global 

Environment”, Proceedings of the ONLINE EDUCA MOSCOW. 1st International Conference on Technology Supported Learning 

and Training; 29 September-1 October 2007; Book of Abstracts, Moscow, Russia: Market DS pp.272-277. 


that offer all of their programmes on-line. It is not accidental that the ideas voiced 
by the leading universities of the Bologna process and the e-Bologna concepts are 
spreading throughout the Western European countries. The goal of the process is to 
reach an understanding and recognition on a global scale based upon the consistent use 
of E-learning tools. 

3.3. Appropriate competitiveness level and access to quality 

E-learning tools are used by the universities in accordance with the pragmatic principle 
of reaching an appropriate level of competitiveness. This is often understood as the 
necessity of providing easy access to education and creating a means for bridging the 
distance gap. If we look at such a large country as Russia, for example, we shall see that 
the only way for those citizens living in the remote regions to get access to high-quality 
education is to take an online (E-learning) course. 

When we speak about the easy access to education and training opportunities, 
we should mention, of course, transborder or transnational higher education. These 
include all types of study programmes, courses, and educational services, including the 
distance learning programmes offered for those learners who study from a different 
country. The study programmes can be part of an education system of a foreign 
country, or can be offered regardless of the characteristics available in a certain 
national education system. 

Some documents related to the mobile education development plans were elaborated 
in accordance with the Lisbon European Council resolutions of 14 December 2000 4 , 5 . 
The plan includes 42 stages that are divided into the following four groups: 

• Steps to improve student and teaching staff mobility (including the steps to 
improve the foreign and native language skills in order to gain better access to 
reliable and helpful information); 

• Steps to ensure financial mobility by concentrating the necessary resources on 
all levels, and steps to ensure easy access to mobile learning for all community 
groups; 

• Steps to diversify and improve the quality of mobile education by introducing 
new forms of provision; by improving the quality of study programmes and data 
resources; by streamlining the structure of the programmes, and by determining 
the status of an associate professor or instructor; 

• Steps to harmonise mobile education programme results by setting a standard for 
periods of study and practical training. 

As the elaboration of transborder education scheme is a direct consequence of the 
internationalisation of the education system, it is closely connected with the new 
ICT use. This is a manifestation of the fact that the education market is becoming 
increasingly global, and this development stage is characterised by the full-fledged 
struggle for market space. There is a lot to fight for. The demand on the world’s higher 
education market grows by 6 % each year, and the growth in demand is much higher 
than the growth in transborder education market. In total, there were slightly more 
than 2 million international students studying around the world in 2003. It is estimated 


4 Presidency Conclusions (2000) Proceedings of the Lisbon European Council; 23 and 24 March 2000; Lisbon, Portugal. 

5 The role of the Universities in the Europe of knowledge (2003), Proceedings of the Commission of the European Communities; 
05.02.2003; Brussels, Belgium. 


that the international student number will be 7.2 to 7.3 million in 2025. The majority of 
the students will be taking transborder courses. 

Education is becoming increasingly transnational because universities in developed 
countries are promoting their traditional study programmes through various means. 
First of all, they found representative offices, branches and even campuses. At least 
75% of all the programmes in transnational education are exported on the franchise 
agreement basis to the overseas branches of universities, and to the representative 
offices of the institutions that offer distance learning courses. In general, transnational 
education programmes are either traditional courses, online courses or integrated 
learning courses. 

The majority of European countries do not have appropriate legal base or by-laws 
to support the exports and imports in the field of transnational education. Only Great 
Britain and Sweden stand out in this respect. They have their own bona fide behavior 
codes and quality assurance standards used to foster the transnational education 
development and to improve its standards. Sweden is the only European country to 
have a clear-cut national policy that describes the transnational education programme 
award recognition rules. There are variants to choose from when it comes to the 
development of national and transnational education providers’ codes of bona fide 
conduct. They can be modeled along the lines offered by Great Britain or Australia 6 . 

Despite the problems, transnational education programmes are being offered and are 
a success. There are a few reasons why it happens. Firstly, the transnational education 
programmes provide additional opportunities for people to have better access to 
education and training programmes with a rather wide range of options. Secondly, if 
transnational programmes are offered on a national market, the traditional universities 
that are present on the market become motivated to compete more for a place on 
the education market with their colleagues, and this helps to improve the learning 
environment. Let’s note that the country exporting the transnational education 
contributes to the improvement of the competitiveness level of the country’s education 
providers, and this is a prerequisite for getting more revenue from the new exports. 

3.4. Integrated access to quality as an indicator of competitiveness 

To ensure access to high quality education is still quite a challenge in Russia, and this 
affects the market situation. 

The challenge is called the digital divide. The term refers to the gap between those 
people who do have, and those who do not have access to ICT (that includes, first of all, 
the Internet access) due to financial restrictions, remote geographic location, lack of 
basic ICT knowledge, etc. This is a problem that many European countries face. Each of 
the OECD countries, for instance, has quite different characteristics as far as the IT use 
is concerned. The characteristics include: the number of students per each PC (which 
range from 5 to 20 in the European countries), the number of students who have a PC 
at home (90% of the total number of students in the Scandinavian countries and up to 
50% in the countries of Central and Eastern Europe), the costs, quality of service and 
access opportunities as far as the Internet use in the universities is concerned. There 
are also various characteristics demonstrated by different educational institutions 
within one country. 

6 The Recognition, Treatment, Experience and Implications of Transnational Education in Central Europe 2002-2003 (2003), 
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Thus, in practice, the introduction of ICT into the learning process can both 
contribute to the improvement of access to education, and sometimes help broaden 
the digital and economic divide. That is why some OECD countries and Russia are 
currently implementing a number of projects in order to provide wider public access 
to ICT use (especially the poor) in universities, libraries and educational centers, and 
in order to offer courses on ICT skills to the teaching staff, and to create incentives to 
businesses to invest into the ICT-supported study programmes for their staff. 

Fortunately, nobody today is trying to see E-learning as a means of providing 
education to the geographically remote entities only and is claiming that E-learning is 
only good as long as it is catering to their needs. 

An interesting approach to quality issues in E-learning is described in Towards a 
Greater Quality Literacy in a E-Learning Europe article by Ulf-Daniel Ehlers 7 , coordinator 
of the European Quality Observatory of the University of Duisburg-Essen. He says that 
“quality is more than just an evaluation at the end of a course. It is a comprehensive 
concept which concerns all areas of E-learning.” He describes three concepts that can 
be combined into a new single concept of quality improvement: education-oriented 
quality development; consideration of all the E-learning process stakeholders’ needs; 
use of a special decision cycle that could help a provider find a unique approach towards 
quality issues. It is necessary to have certain competence to do this, not just tools for 
doing the work. He adds that “because of its open nature and of involving stakeholders 
into the process of quality development the approach can be called participatory 
approach to E-learning quality.” 

3.5. Integrated access: From chaos to quality 

E-learning course efficiency is seen best if a higher education institution offers not only 
face-to-face or distance learning courses but a combination of the two. 

Such a model can be used in the most efficient way, if a university offers a concept 
of integrated learning where the university management can find an optimised 
combination of face-to-face learning, virtual campus online and courses offline, and 
integrate the multidiscipline learning content and place it on a single data carrier. In 
this case, the main E-learning tools cannot be regarded as makeweights that are added 
merely to make the traditional study programmes seem acceptable. They become a 
robust tool that helps to acquire important competence, skills and knowledge that can 
be used in practice. 

It is becoming evident that Internet connection in an educational institution is 
more than merely a means of transferring learning material from one point to another, 
which serves the students and the teaching staff. Being a tool supporting an E-learning 
programme, the Internet is most efficiently used when it is used as a learning 
environment and not only as a means of byte traffic transferring. The E-learning system 
is pragmatic, not because E-learning courses are sold and bought with e-money, but 
because the opportunities that the Internet provides are used as a means to support the 
use of teaching methodology. 

The management of the most innovative universities is now asking a question, for 
example, of whether an E-learning course could be a substitute for a live contact course 
with an instructor. In theory, any student who takes a course for an external degree 

7 Ehlers U.D. (2006) ‘O noBbimeHUH rpaMOTHOCTM b Bonpocax KanecTBa b cc^epe E-learning b EBpone’ [Towards a Greater Quality 

Literacy in E-Learning in Europe], Vyisschee Obrazovanie v Rossii 12: p.43-53. 


does not have a live contact with an instructor, but examines the various information 
sources available. An E-learning course, even if devoid of any human contact, 
undoubtedly turns a classical external degree course into an integrated learning course. 
However, the absence of human contact in practice has given reasons to believe that 
E-learning is extreme learning (refers to not structured learning, the positive sides of 
which are not visible). Elliot Masie, a pioneer of American E-learning, was the first one 
to coin the witty expression 8 . Is it reasonable to use extreme learning in an E-form? 

The best solution would be not the substitution of a live contact course, but the use 
of a combination of an experienced live instructor skills and high-tech tools. The tools 
can be used by the E-learning tutors who should stimulate the students’ activity. If such 
a combination is used, a systemic role shift occurs and the learners turn into E-learning 
partners 9 . 

However, a different answer to the question is also possible if two things occur. 
Firstly, E-learning instruments could undoubtedly become a good alternative tool for 
not-quite-competent instructors who are in the habit of relying on the obsolete data 
taken from the old sources. E-learning is efficient as long as the learning content is 
updated regularly and the teaching methodology helps to master the learning material 
and to acquire knowledge. Competition between the content providers in an open 
learning environment is fierce, unlike the competition between the instructors who 
teach in the university classrooms behind closed doors. 

Secondly, in some cases the full or temporary absence of contact with a live 
instructor, and studying in a virtual environment is important for forming such 
competences as the ability to process large volumes of data and extract the essential 
information, the ability to put knowledge into practice, the ability to participate 
in teamwork and the eagerness to learn more. This is observed, for example, when 
students do the on-the-job training, or participate in business games in simulated 
virtual business environments where there could be no instructor even in theory. 
Typically, the attention of employers is drawn to such competences in the applicants 
who are one of the categories of the stakeholders who are present on the market. 

The other three categories of stakeholders interested in education quality are: State, 
academic community (universities, including this university) and students. 

3.6. Quality of access 

The evolution of our vision of the role of E-learning as a system that is being introduced, 
and the place it occupies as a tool that helps to foster the introduction of innovations 
in the field of education, runs in parallel with the evolution of our vision of the place 
it occupies in the relevant quality management system. The global education market 
trend is moving from quality evaluation towards quality management, and further, 
to quality assurance offers. This is the field where the interests of the international 
accreditation agencies are focused 10 . 

European quality evaluation systems used in the field of education differ from one 
another in terms of goals, objectives, procedures and criteria, and in terms of the way 
the state executive bodies, community organisations and professional communities are 

8 Masie E. (2001) ‘TeachLearn TRENDS: E-Learning, Training and E-Collaboration Updates’, Newsletter Nr. 197, 26 February. 

9 Fetterman D., Wandersman A. (2005) Empowerment Evaluation, Principles in Practice. N.Y. - London 

10 Hamalainen K., Hamalainen K., Dorge Jessen A., Kaartinen-Koutaniemi M., Kristoffersen D. (2003) Benchmarking in the 
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involved in the process. Therefore, in the environment where “the universities' dual 
role - convergence as much as divergence from a national consensus - does not indeed 
make easy the relations with the powers” 11 , European countries try to foster harmony 
between different evaluation systems available, and put the internal quality assurance 
systems of universities in a position where they will be fully in charge of the quality 
assurance measures taken. Besides, the national education systems are being reformed, 
and the general reform trend is to decentralise the power structure of state executive 
bodies, broaden the scope of university authority, and give more executive powers to 
universities, the community and public organisations. 

In Russia, the state education authorities support the development of the external 
quality evaluation system in order to make the education system change faster and 
improve the quality of the provision of education as soon as possible. That is why the 
external quality assurance mechanisms have begun to take new shape recently. The 
employers’ associations have started elaborating the professional standards, various 
market sector representatives have started elaborating the qualification requirements 
for the graduates to meet, the rating agencies monitor the market where higher 
education institutions offer their services and examine the study programmes, and the 
Russian expert associations offer audit and accreditation services to universities. 

The support of the state education authorities, rendered to the development of the 
education quality evaluation system, inevitably brings about the interaction between 
the state evaluation system and a system supported by non-governmental agencies. 

Such interactions are based on the procedures and criteria elaborated by, and the 
findings shown by the accreditation agencies in the course of the state education 
authorities’ preparation of a final accreditation statement 12 . 

3.7. Quality of integrated access 

As E-learning is to a large extent a product of the evolution of technology, the 
phenomena that can be called ‘technological determinism’ are observed in Western 
European countries and in the USA where the relevant stakeholders are trying to create 
a platform for setting the common standards in the field. Such data exchange standards 
as IMS, SCORM, LOM/LRM or the IEEE LTSC standards are sometimes treated as the 
E-learning standards. In Russia, on the contrary, the state’s educational standards deal 
with the study programme content only. They do not prescribe the basic technology or 
the methodology for the teaching process. The EFQM and ISO 9001 standards have 
recently been regarded by some as an alternative to the state’s educational standards. 
However, as Leopold Kause and Christian Strake reasonably state, these standards 
describe the administrative processes that are typical for any company and do not 
include provisions where the learning process specifics, and, E-learning process 
specifics, in particular, are described 13 . The ISO 19796-1 standards that are being 
elaborated on the basis of the above standards would probably become an E-learning 
standardisation tool. 

The European Foundation for Quality in E-learning (EFQUEL) focuses on the quality 
evaluation, quality management and quality assurance in the field of E-learning. Today, 
the Foundation, established in 2005, sees the development of the quality assurance 

11 Academic Freedom and University Autonomy Parliamentary Assembly of the Council of Europe, 2nd June 2006 Recommendation 
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system and the implementation of quality standards in E-learning as its main goal. 

The European Commission Directorate General for Education and Culture supported 
the EFQUEL foundation. Currently, the Foundation brings together some of the most 
significant European actors in the field of E-learning. The Russian Agency for Higher 
Education Quality Assurance and Career Development (AKKORK) has been one of the 
Foundation’s full members since the date it was founded. The audit programmes offered 
by AKKORK are aimed at ensuring the competitiveness of the study programmes 
and other services offered by the universities. When conducting the audit, AKKORK 
examines the graduate knowledge level at the end of the study period, using an 
internationally recognised competence-based approach toward auditing in its work. 
AKKORK is an associate of the European Association for Quality Assurance in Higher 
Education (ENQA) 14 , and a full member of the International Network for Quality 
Assurance Agencies in Higher Education (INQAAHE), and the Asia-Pacific Quality 
Network (APQN). AKKORK, together with the EFQUEL, takes part in the European 
University Quality in E-learning (UNIQUe) 15 project in Russia. The goal of the UNIQUe 
project is to establish a pan-European accreditation system for traditional universities 
where E-learning tools are used. The most prominent participants in the UNIQUe 
project are granted the EQuality Award for being the best providers of ICT-supported 
learning. 

The idea behind the UNIQUe project is to recognise the rise of integrated learning 
as a historical fact in the world’s education system. A university is expected to use the 
integrated learning system if E-learning courses occupy at least 20% of the total time 
dedicated for the provision of study programmes. 

The UNIQUe project has become a proving ground where various approaches 
are being tested towards the quality assurance system evaluation schemes used by 
the I-learning and E-learning users. In the EFQUEL Forum (Helsinki, Finland) on 
24 September 2009, six traditional Western European universities and one Eastern 
European university - Moscow University of Industry and Finance - were awarded the 
UNIQUe ‘E-learning quality label’ for 2009-2012. 

The main area where the EFQUEL is going to play the role of a consolidator is the 
improvement of the quality of provision. New data use, stimulation of new data use 
and the principle formation for new I-learning partner relations can help improve the 
quality. The learning content and technology used in E-learning help offer the high 
quality of provision, as the teaching process is not devoid of the influence of the human 
factor. 

The fact that such goals are declared, cannot be disrespectfully ignored, as until 
recently, there have been no internationally accepted theories or concepts that would 
go in line with the practical requirements, and that could help improve the quality of 
provision with the help of the use of E-learning. Similarly, there is no comprehensive 
E-learning quality evaluation system that would really work and take into consideration 
all the main functional aspects and introduction characteristics of the system. 

In the final analysis, the EFQUEL, in cooperation with the academic community 
members and all the education market players, is trying to establish a solid quality 
assurance system in Europe for E-learning providers that would integrate quality and 


14 European Association for Quality Assurance in Higher Education (2005), Helsinki. 
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accessibility and help the traditional universities, that predominantly offer face-to-face 
learning programmes, make the necessary internal transformations. 

3.8. Monitoring access to quality and conclusion 

The key point here is the understanding of the fact that the distance learning 
programmes of universities worldwide are provided by the distributed networks of 
campuses. The potential positive development of E-learning is clearly visible if used by 
the distributed network providers. Such networks are the best place for I-learning to be 
developed, as they provide access to high-quality education at an affordable cost. 

As for modern Russian universities, for example, the only legally supported 
operational base for them is the branch network. Such branches offer officially 
recognised distant E-learning programmes only. 

Meanwhile, European universities that use E-learning to provide programmes of 
any type have no such restrictions. They have the right to offer online courses and 
consider their students’ home and office computers as access points. In fact, they use 
the Internet as their supporting network, not as a separate learning environment. Such 
remote training centers can be called ‘branches’, or something else, and can be located 
in any country in the world. It is crucial here that the quality standards of the branch 
for provision of education be in line with the quality standards for integrated learning, 
set by the head office. 

If appropriate Russian state education management bodies give their consent to the 
setting of access points, they reserve the right to monitor the use of the operational 
technology, access and transferred content characteristics of these points. The points 
enable remote operations for universities and help them to interact with distant 
E-learning students while ensuring appropriate quality of course provision. 

The main goal of the state education authorities is to check whether university 
graduates have the necessary professional competences when they leave the university. 
In this case, distant course providers should take efforts to improve the quality of their 
courses with comparable professional competences and knowledge as face-to-face 
course providers do. E-learning is undoubtedly an efficient tool that can be used to 
reach this goal. By using this tool, distance learning students have access to the high- 
quality education that they need, and that the state education authorities approve of. 
The university head office should be in charge of the E-learning course provision. The 
fierce competition on the regional markets will undoubtedly create an environment 
where universities would improve their competitiveness level to enter the world market. 

In European countries as well as in Russia, the external evaluation of the quality of 
course provision in the field of education should be provided by accreditation agencies. 
Such evaluations are necessary in order to protect the legitimate interests of the market 
actors and the interests of the local community 16 . That’s why the issue of independence 
is regarded as being critically important and deserving of special attention, since it 
forms, among others, the basis for professionalism, and thus, the basis for trust . 17 The 
quality evaluation of E-learning programmes and the evaluation system for university 
practices of should be based on the fair principles of transparency in accreditation and 
external quality evaluation, and in the self-evaluation of the university. 


16 Eaton J.S. (2001) Distance Learning: Academic and Political Challenges for Higher Education. CHEA: Washington D.C. 
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It is therefore necessary to take the E-learning phenomenon into consideration when 
a correlative scheme with achievement descriptions for various educational institutions 
is being created. It is clear that the quality assurance efforts that are not in line with 
the general notion of E-learning, and that do not include the university evaluations 
based on the evaluations of the university’s E-learning prospects, cannot guarantee the 
stakeholders that the institution provides E-learning programmes in accordance with 
a code of good practice. The fact of using the innovative potential of E-learning should 
be regarded separately as one of the criteria to be utilised when learning environment 
characteristics are assessed. In this way, the efforts of the abovementioned universities 
would not be ineffective. The comprehensive evaluation of the quality of course 
provision, and the quality assurance systems that can be used to assess E-learning 
programmes have their clear advantages too. They can be used on an international 
scale as well. 


Chapter 4: 

Challenges for quality assurance 
organisations: The case of NVAO 

Fred Mulder, NVAO, the Netherlands 


4.1. Summary 

After briefly introducing the Accreditation Organisation of the Netherlands and 
Flanders (NVAO), this article presents NVAO’s position on the four policy issues for 
Quality Assurance agencies discussed in the Swedish National Agency for Higher 
Education’s (NAHE) report E-learning quality: Aspects and criteria for evaluation of 
e-learning in higher education 18 . 

4.2. NVAO 

NVAO (in Dutch: Nederlands-Vlaamse Accreditatieorganisatie) is the Accreditation 
Organisation of the Netherlands and Flanders, which is part of Belgium. The 
organisation was established by an international treaty. NVAO is based in The Hague, 
in the Netherlands. NVAO is a full member of ENQA, and is listed on the European 
Quality Assurance Register (EQAR). 

NVAO’s mission is to independently ensure the quality of higher education in the 
Netherlands and Flanders by assessing and accrediting programmes, and to contribute 
to enhancing this quality. Thus far, this accreditation system has operated at the level of 
programmes, not institutions. 

In addition, NVAO contributes to raising quality awareness within higher education 
and advancing the position of higher education in the Netherlands and Flanders in the 
national and international context. 

Its core business is accrediting all existing Bachelor’s and Master’s degree 
programmes in these two countries. Also, NVAO assesses proposals for new degree 
programmes that universities intend to launch. The actual assessment is done by expert 
panels. On the basis of their assessment report, NVAO takes a yes or no decision. 

4.3. Policy Issue 1: Integration of e-learning criteria in the national quality 
assurance system 

NVAO is not concerned with the quality of e-learning as such; the NVAO accreditation 
framework does not feature any explicit reference to it. What we are concerned with 
is the quality of Bachelor’s and Master’s degree programmes, which may or may not 
involve elements of e-learning. 

In its discussion about the integration of e-learning criteria in the national quality 
assurance system, the NAHE report formulates this conclusion: “In order for quality 
assessment of e-learning to become an integral part of national quality reviews, aspects 
and criteria need to be incorporated into the general basis for assessment.” 

18 Swedish National Agency for Higher Education. (2008). E-learning quality: Aspects and criteria for evaluation of e-learning in 
higher education. Report 2008:11 R. 


NVAO agrees with this statement, although our interpretation of “incorporated” may 
not conform completely to the one intended by the NAHE report. We are convinced 
that incorporating e-learning specific aspects and criteria does not necessarily have 
to entail a revision of our assessment framework. NVAO’s assessment framework is 
an “open” one, and is very well capable of accommodating input on e-learning. For 
example, if teachers have been trained to become versed in the art of on-line tutoring, 
it is possible - and relevant - to elaborate on this under the assessment frameworks’ 
“Quality of Staff” standard. This is just an example. As for a more systematic and 
comprehensive approach, we do not prescribe or impose anything. If interested 
parties consult us on how to feed evidence and considerations on e-learning into the 
accreditation process, we refer them to the benchmarking statements that have been 
developed by the E-xcellence project 19 . Launched in 2005 by the European Association 
of Distance Teaching Universities (EADTU), E-xcellence has evolved into an attempt to 
work within existing national QA frameworks rather than re-inventing them. NVAO 
has participated from the start and has also been involved in the E-xcellence+ follow-up 
project (2008-2009). 

How about the correspondence between the E-xcellence statements and the 
component parts of the NVAO accreditation framework? At first glance, they appear to 
be totally unrelated. But scrutinised more closely, the 33 E-xcellence statements and the 
21 NVAO quality standards yield an underlying commonality enabling one to interpret 
almost all of the statements as matching one or several NVAO standards. 

For instance, let us consider the E-xcellence statement number 19, the first one in the 
section on course delivery, which reads as follows: 

“The technical infrastructure maintaining the e-learning system should be fit 
for purpose and support both academic and administrative functions. Technical 
specification should be based on a survey of stakeholder requirements and involve 
realistic estimates of system usage and development.” 

In our view, this statement can be interpreted to match the following two NVAO 
standards: 

• “The accommodation and material facilities are sufficient to implement the 
programme.” [Theme: Facilities and Provisions; standard: Material facilities] 

• “Staff, students, alumni and the relevant professional field will be actively 
involved in the internal quality assurance system.” 

• [Theme: Internal Quality Assurance; standard: Involvement of staff, students, 
alumni and the professional field] 

The first part of the E-xcellence statement is about the quality of the technical 
infrastructure maintaining the e-learning system, while the first NVAO standard is 
about material facilities in general. The second part of the E-xcellence statement can 
be interpreted as an expression of need for technical infrastructure when consulting 
relevant parties in the context of internal quality assurance, which, again, features in 
the NVAO framework in a general fashion. 

The above is just one example of the correspondence between the E-xcellence 
benchmarking statements and the NVAO assessment framework’s standards. 


19 www.eadtu.nl/e-xcellence 


Admittedly, the two are somewhat loosely coupled, and it therefore takes a certain 
effort to interpret the one in terms of the other. In our view, however, it can be done 
- albeit with a few exceptions - and it is worth it. NVAO is currently in the process of 
further exploring this correspondence. 

4.4. Policy Issue 2: Intelligence and competence within the organisation 

The NAHE report states that incorporating e-learning specific aspects and criteria 
into the general basis for assessment requires intelligence and competence within 
the organisation: “A special function for e-learning needs to be set up within the quality 
assurance agency, i.e. a function with the task of monitoring, on a continuous basis and 
under special regulations, national and international developments within e e-learning. 

To keep pace with international developments, we recommend the adoption of a strategy 
for extended representation in international organisations, projects and networks. The 
establishment of an e-learning advisory board is also recommended.” 

As with the previous policy issue, NVAO agrees, but what we consider crucial is that 
e-learning specific intelligence and competence is present among panels of experts. 

In the case of reaccreditation of existing degree programmes, external assessment 
is done by a panel of experts composed by and run by an Assessment Agency. Their 
report is the material that NVAO has to base its accreditation decision on, by validating 
the expert panel’s assessment. In cases where e-learning does not feature properly in 
an assessment report, NVAO is entitled to request additional information or a revised 
report. But this is not an ideal situation. In the near future, our protocol for Assessment 
Agencies will be renewed, and we will definitely include the obligation of having an 
e-learning expert in a panel where this is required by the character of the programme 
to be assessed. 

In the case of initial accreditation of new degree programmes, NVAO itself composes 
and runs the panel of experts. That way, we are in a position to actively ensure the 
presence of e-learning expertise in a panel. 

It is obvious that the above requires a measure of e-learning specific intelligence and 
competence within NVAO. Two of our staff is reasonably up to date with e-learning 
developments and we do our best to keep pace with international developments. 

4.5. Policy Issue 3: Cross-boundary education changes the conditions for 
quality assurance 

“Knowledge exchange and cooperation between quality assessment agencies and 
organisations across national borders are necessary in order to harmonise and safeguard 
quality assurance strategies and policies”, according to the NAHE report in its conclusion 
regarding cross-boundary education from an e-learning perspective. 

The ENQA/NAHE workshop “Quality Assurance of E-learning” (Sigtuna, Sweden, 

7-8 October 2009) was a fine example of knowledge exchange and will hopefully serve 
as a starting point for cooperation. 

NVAO’s preferred approach of cross-boundary education, whether or not of an 
e-learning nature, is through recognition of the accreditation decision made by the 
national accreditation organisation of the country or countries involved. 

We do not recognise just anybody’s accreditation decisions. Rather, we do recognise 
the accreditation decisions made by our partners in the European Consortium on 
Accreditation in higher education (ECA) 20 . 


20 www.ecaconsortium.net 


NVAO is fully aware that this arrangement is not up to the task of assessing the 
quality of the education involved when students combine courses provided by, say, five 
universities on three continents. Thus, admittedly, we still have a long way to go. 

4.6. Policy Issue 4: Methodological development 

The NAHE report’s fourth policy issue is about methodological development. The 
conclusion reads: “Extensive methodological development will be necessary to adapt the 
general methods for assessment of quality in higher education to the assessment of quality 
in e-learning.” 

It could be argued that methodological development shows substantial overlap 
with policy issues 1 and 2. However, the NAHE reports’ discussion of this final issue 
contains at least one important element that has not been touched upon earlier: “While 
the assessment of digital applications implies difficulties for the assessors, they also offer 
novel opportunities for them to reach the heart of teaching and learning, as many of the 
educational environments are more easily accessible.” 

In NVAO’s view, a true e-learning expert is someone who is fully aware of the novel 
opportunity of having access to the record of learning interaction and engagement. 

Thus, we say yes to methodological development, and, again, as with policy issue 
number 2, it is the external experts who have to do the job. They must embody sound 
methodology and represent methodological development. It is NVAO’s duty to ensure 
that panels are adequately composed in this respect. 


32 


Chapter 5: 

Benchmarking eLearning in higher 
education 

Findings from EADTU’s E-xcellence+ project and 
ESMU’s benchmarking exercise in eLearning 

Ebba Ossiannilsson, Lund University, Sweden and Oulu University, Finland 21 


5.1. Introduction 

The ENQA workshop in Sigtuna, Sweden, in October 2009, focused, among other 
issues, on the eLearning Quality Model (ELQ) (NAHE, 2008) by the Swedish National 
Agency for Higher Education (NAHE), and on their survey on quality assurance of 
eLearning in higher education on a European level in nine selected countries. One 
conclusion from their survey was that quality in eLearning is a non issue for many, 
especially for quality assurance agencies. There are promising projects managed by 
organisations such as the European Association of Distance Teaching Universities 
(EADTU), the E-xcellence initiative, the European Foundation for Quality in E-learning 
(EFQUEL), the UNIQUe initiative, the Joint Information Systems Committee (JISC), 
the Higher Education Academy (HEA), and the Pick&Mix benchmarking methodology. 
However, all these projects deal with eLearning as a separate issue. In the ELQ-model, 
it was suggested that the same criteria for quality should be applicable to eLearning as 
it is to traditional campus-based education (ENQA, 2009). The aims of the workshop 
were to discuss new aspects and criteria for eLearning to be considered and to discuss 
on how to integrate eLearning criteria in national evaluation programmes. 

Below briefly follows the meaning of the concepts ‘benchmarking’ and ‘eLearning,’ 
as this article aims to focus on experience in benchmarking eLearning in higher 
education. In addition, the above-mentioned aims of the ENQA workshop will be 
reflected and discussed. 

Benchmarking is a common method in quality assurance and enhancement, and 
frequently used in different sectors. It has gradually become a common method 
even in higher education. Of course, benchmarking and quality aspects are seen 
differently due to the interpretations and understandings of the concept. However, 
it is not yet common to benchmark eLearning in higher education (Ossiannilsson, 
2010a). Moriarty (2008) defines benchmarking as “an exemplar-driven teleological 
process operating within an organisation with the objectives of intentionally changing an 
existing state of affairs into a superior state of affairs (p. 30).” Moriarty & Smallman 
(2009) further express it as follows: “The locus of benchmarking lies between the 
current and desirable states of affairs and contributes to the transformation process that 
realises these improvements” (p. 484). The European Centre for Strategic Management 

21 The author would like to thank the Swedish National Agency, former NSHU, for the initiative to include Lund University in the 

EADTU E-xcellence+ project. Additionally, the author would like to express her thanks to EADTU, ESMU and the partners in the 

projects for fruitful co-operation during the exercises. 


(ESMU) defines is as “an internal organisational process which aims to improve the 
organisation's performance by learning about possible improvements of its primary and/or 
support processes by looking at these processes in other, better-performing organisations” 
(van Vught, et al., 2008a, p. 16). According to the above-mentioned definitions, 
benchmarking is about change. 

The concept of eLearning is not very easy to interpret or define (COM, 2001), 
and is often seen as a concept covering the delivery of learning, training or education 
programme by electronic means. While blended learning is eLearning combined with 
other, more traditional training methods, eLearning might be used as a prequel or 
sequel to face-to-face events (Clinch, 2005). In this article, eLearning and blended 
learning are discussed as more or less synonymous concepts. 

The background and theoretical perspectives of this article are based on findings 
from Lund University’s participation in and implementation of the benchmarking 
initiatives by EADTU’s E-xcellence + 22 (Ossiannilsson & Landgren 2010a, b, c; 

Ubachs, 2009) and the eLearning benchmarking exercise (ELBE) by ESMU in 2009 23 
(Ossiannilsson & Landgren 2010a, b,c; Willimas & Rotheram, 2010). These perspectives 
and findings include national, regional, European and international networks 
and perspectives, discourse in the area of eLearning, current research results in 
benchmarking eLearning in Higher Education Institutions and its implications, values 
and impacts (Ossiannilsson, 2010a, b). 

5.2. Reasons and benefits for Lund University to participate in European 
benchmarking exercises 

During 2008-2009, Lund University took part in two European benchmarking 
initiatives, E-xcellence+ by EADTU (Ossiannilsson & Landgren 2010a, b, c; 

Ubachs, 2009), and ESMU’s eLearning benchmarking exercise ELBE (Ossiannilsson 
& Landgren 2010a, b, c; Williams & Rotheram, 2010). E-xcellence+ was carried 
through by an initiative from NAHE, former NSHU (Swedish Agency for Network and 
Co-operation), as an attempt to try the tool not only in further initiatives, but also in a 
traditional university for the identification of challenges and benefits. 

EADTU’s E-xcellence+ operates largely on an individual basis. The methodology was 
to conduct an online QuickScan, a full assessment and site-visits with experts, and to 
work out road maps. The benchmarks were grouped in six areas: strategic management, 
curriculum design, course design, course delivery, staff support and student support, 
covering in total 33 benchmarks with indicators and excellence level. They can also 
be clustered in three main headings: management (strategic management), content 
(curriculum design, course design, course delivery) and services (staff support and student 
support). A full online manual was available. When E-xcellence+ was carried out at 
Lund University, the focus was mainly on two of the International Master programmes: 
LUMA-GIS, a master course on geographical information systems (Pilesjo, 2010), and 
IIIEE, a Master programme on Environmental Science (IIIEE, 2010). The exercise was 
also carried out at Lund University’s central administration levels with eLearning and 
e-resourcing responsibilities. 

Fortunately, the two programmes at Lund University, LUMA-GIS and IIIEE, which 
were involved and examined, got the EADTU’s E-xcellence Associates label (Figure l). 24 


22 E-xcellence, http://www.eadtu.nl/e-xcellenceplus/, available 100721 

23 Benchmarking eLearning in European universities, http://www.esmu.be/benchmarkingelearning.html, available 100721 

24 E-xcxellence Associates, http://www.eadtu.nl/e-xcellencelabel/, available 100803 


The E-xcellence Associates focus on the improvement of four primary elements of 
progressive higher education: accessibility, flexibility, interactiveness and personalisation 
(Figure 2), (EADTU, 2008). All these fundamental success factors were reached to 
a high extent in the benchmarking. The label led to recognition, not only locally at 
the university, but also regionally, nationally and internationally with several benefits 
for students, such as reception of grants, international co-operation and networking. 
Responsible staff for the LUMA-GIS was awarded the Lund University pedagogical 
prize of 2009 for their innovative and good quality work, and contribution to 
internationalisation within the LUMA-GIS programme, as shown below. 



Figure 1. The EADTU E-xcellence Associates' label to LUMA-GIS and IIIEE at Lund University, 
Sweden. 
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Figure. 2. Universities and University Programmes that were awarded the E-xcellence 
Associates label belong to the EADTU Associates in Quality. 

Unlike EADTU’s E-xcellence+, the ELBE benchmarking operates more on a 
collaborative basis. It has co-operated, however, with the former, and EADTU’s Quick 
Scan was used as a starting point for the ESMU exercise with the participation of 
nine European universities. The benchmarking criteria were based on the six areas 




of EADTU’s E-xcellence+, as mentioned above, and updated according to EADTU’s 
evaluation to suit the partnership. The ESMU benchmarking was carried out only at 
a central administration level at Lund University (the same as mentioned above). In 
addition to the Quick Scan, the a assessment was done and two workshops were carried 
through. The establishment of roadmaps was the final step in this benchmarking 
process (Ossiannilsson & Landgren, 2010a, b, c; Williams & Rotheram, 2010). 

During 2009-2010, Lund University participated in the first dual-mode distance 
learning benchmarking club, based on the Pick&Mix methodology (Bacsich, 2009a, 
b, c; Ossiannilsson & Landgren, 2010a, b, c). One task was to look for concordances 
within Pick&Mix, EADTU's E-xcellence+ and ESMU's ELBE, and to use the results 
to suggest new or redefine the existing Pick&Mix benchmarks. Interesting remarks 
and suggestions for further development of the method have been given, such as 
creating new benchmarks, updating the terminology, making it more student-centered, 
and including blended mode perspectives (Ossiannilsson & Landgren, 2010a, b, c). 
Additionally, a study on different benchmarking models reveals an emerging contextual 
framework with a holistic approach. 

In summary, Lund University’s participation in the European benchmarking 
initiatives show that there are at least ten good reasons to participate in benchmarking 
exercises: self-assessment of the institution; better understanding of processes; measuring 
and comparing; discovering new ideas; obtaining data to support decision-making; setting 
targets for improvement; strengthening institutional identity; strategy formulation and 
implementation; enhancing reputation; responding to national performance indicators 
and benchmarks; and setting new standards in the contexts of higher education reforms 
(van Vught et.al., 2008a, b). The benefits were: participation in internal processes; 
involvement of faculty or department staff; and transparency of courses, the entire 
educational process and basic issues (i.e. documents, Web-pages, policies, and other 
such issues). The benchmarking exercise was process orientated, and served as a 
learning exercise in working with critical colleagues during site visits and formulating 
roadmaps for the implementation of desired changes according to the universities’ 
benchmarking process results. Through the benchmarking process, the strengths and 
weaknesses of each institution and/or programme became obvious. 

5.3. The Swedish context and Lund University 

Below follows a short description of the Swedish context, and some current initiatives 
from Lund University. 

The Swedish Net University Agency, launched in 2001, was a Swedish government 
agency coordinating, promoting, supporting, developing and marketing IT supported 
distance learning courses in higher education. 35 universities and university colleges in 
Sweden offered distance learning courses through a national and international portal, 
the Swedish Net University. The Swedish Net University Agency was not a university in 
itself, and thus, did not offer distance courses or programmes of its own. The agency 
offered a database of courses and programmes from the universities. Most of the 
courses were, however, taught in Swedish, although some courses were available also in 
English. Like all publicly funded higher education in Sweden, there was no tuition fee. 
The agency also supported the development of distance learning courses, and promoted 
the exchange of experience in the held. When the Net University was closed down in 
2006, the Agency for Networks and Co-operation in Higher Education (NSHU) took 


over the role as a co-ordinating national body. 25 NSHU’s overarching goal was that 
more people would have access to higher education, not only undergraduate students at 
universities or university colleges, but also those who had completed their studies and 
were either working or continued on to graduate studies. NSHU was closed down in 
2008, and more responsibilities were dedicated to single universities in the country. 

Lund University was among the first to adopt eLearning in Sweden in the early 90s, 
and well known internationally for its innovative and early approach. Lund University 
was one of the largest and most active partners of the Swedish Net University during its 
operating years (LU, E-xcellence, 2007-2008; LU, ESMU, 2009). 

The Strategic Plan for 2007-2011 (LU, Strategic Plan of Lund University, 

2006) emphasises four areas: quality assurance, cross-disciplinary collaboration, 
internationalisation, and leader, teacher and employee excellence. All areas are related 
to eLearning, mirroring how technology can be used and developed to facilitate 
learning, teaching and research. Four prioritised areas are dedicated in the context 
of the strategic plan: 1) attractive study and learning environments [including Virtual 
learning environments, eLearning, Learning Management Systems (LMS) and Personal 
Learning Environments (PLE)]; 2) strong research and innovation environments; 

3) increased transparency and 4) more prominent infrastructure. Again, all four 
areas are related to eLearning or blended mode strategies. A self evaluation project on 
education, the EQ11, was launched at Lund University in 2010 (LU, Education Quality, 
2011), and is expected to complement the new national quality assurance by the 
Swedish National Agency (NAHE, 2010a) in the coming years. The EQ11 focuses on 
attractive learning environments in education, both physical and virtual, and concerns 
outcomes, alignment, leadership and management, scholarship, internationalisation, 
cross-boundary activities and innovation. 

The Strategic Plan for 2007-2011 (LU, Strategic Plan of Lund University, 

2006) emphasises four areas: quality assurance, cross-disciplinary collaboration, 
internationalisation, and leader, teacher and employee excellence. All areas are related 
to eLearning, mirroring why and how eLearning, or enhanced technology for learning, 
teaching and research, can be used and developed to facilitate learning. Four prioritised 
areas are dedicated in the context of the strategic plan: 

1. attractive study and learning environments (including Virtual learning 
environments, eLearning, Learning Management Systems (LMS) and Personal 
Learning Environments (PLE); 

2. strong research and innovation environments; 

3. increased transparency and 

4. more prominent infrastructure. 

Again, all four areas are related to eLearning or blended mode strategies. A self 
evaluation project on education, the EQ11, was launched at Lund University in 2010 
(LU, Education Quality, 2011), and is expected to complement the new national quality 
assurance by the Swedish National Agency (NAHE, 2010a) in the coming years. The 
EQ11 focuses on attractive learning environments in education, both physical and 
virtual, and concerns outcomes, alignment, leadership and management, scholarship, 
internationalisation, cross-boundary activities and innovation. 


25 NSHU former homepage, main areas, http://www.nshu.se/english/page/4683/mainareas.htm, available 100730 


eLearning and blended learning are among prioritised areas to be developed at 
Lund University under the umbrella of one of the strategic areas, the attractive study 
environment The infrastructure for eLearning and blended learning emphasises 
transparency, accessibility and independence from geographical time zones and 
location. The objective is to meet students in different geographical locations and share 
their contributions to their learning environment and education in the 21st century 
(i.e. the “new millennium learners,” or the “net generation”). All policies and plans are 
based on goals with student involvement, students’ rights, democracy, transparency 
and accessibility. Some 4000 online students are enrolled each semester, and some 100 
courses are available at Lund University in 20101. 26 

The action plan for the ELBE project (LU, ESMU, 2009) stated that Lund University 
should meet the following goals: 

1. The strategic management goal to meet students from any location, use and work 
with their knowledge and skills, and prepare them for the professional and the 
academic arena in order to contribute to a democratic and sustainable global 
development within the society in the 21st century. Further objectives are to 
encourage widening recruitment and lifelong learning, to facilitate attractive 
study and learning environments both on Campus and virtually, and to increase 
and encourage strong development of leader, teacher and employee excellence. 

2. The Curriculum design. Course design and Course delivery goal, of which the 
actions include programmes and courses that are continuously quality assured 
through validation processes, and planned according to students’ demands, 
scientific developments and community needs. 

3. The Staff support goal according to which all Lund University teachers should be 
able to participate in Compulsory Higher Education Teacher Training (CHETT), 
which is a national standard (with 10 weeks at a minimum) since 2005, and 
should be until 2015 (Lindberg-Sand & Sonesson, 2008). In addition, a greater 
focus was brought on scholarships for teaching and learning (Kreber, 2002a, b; 
Trigwell & Shale, 2004), constructive alignment (Biggs, 2003) and encouragement 
of pedagogical qualification and recognition. 

4. The Student support goal was aimed to encourage students to get involved in, 
and to have access to e-infrastructure, (i.e. e-resources and the e-Library that are 
independent of place, time and languages). In addition, stronger enhancement 
of carrier development in the area of innovative pedagogy, (i.e. the use of new 
technology) was expressed as an important objective. 

At Lund University, several initiatives have recently been, and are currently being 
carried out, among them six international interdisciplinary eLearning master 
programmes that were developed during 2007-2008 (Nilsson & Ossiannilsson, 

2008). The Framework of these courses take into account concepts of student learning 
experiences and how they are related to different methods and media according to the 
models of Laurillard (2002) and Clinch (2005). Laurillard identifies five media forms: 
narrative, interactive, communicative, adaptive and productive. She argues that different 
media forms have different affordances, (i.e. they provide a different level of support for 
various kinds of learning experiences). According to Conole and Fill (2005), narrative 
media tell or show the learner something (eg. text, images). Interactive media respond in 


26 Net-based courses and programmes at LU2010, http://www.lu. se/o.o.i.s?id=320&func=distance, available 100730 


limited way to what the learner does (e.g. search, engines, multiple choice tests, simple 
models. Communicative media facilitate exchanges between people (e.g. email, discussion 
forum). Adaptive media change by what the learner does (e.g. some simulations, virtual 
worlds). Productive media allow the learner to produce something (e.g. word processor, 
spreadsheet) 27 (see Table 1). 

Table 1. Table on learning experiences, methods and media forms by Peter Clinch (2005), 28 
based on Laurillard (2002). 


LEARNING EXPERIENCE 

METHOD/TECHNOLOGIES 

MEDIA FORMS 

attending, apprehending 

print, TV, video, DVD 

narrative 

investigating, exploring 

library, CD, DVD, Web resources 

interactive 

discussing, debating 

seminar, online conference 

communicative 

experimenting, practising 

laboratory, field trip, simulation 

adaptive 

articulating, expressing 

essay, product, animation, model 

productive 


Furthermore, at the Faculty of Law of Lund University, courses have been developed 
with eLearning since the m id-90 's with large success, and since then, they have 
often been ranked among the most popular courses. Another initiative to support 
widening recruitment and lifelong learning at the Faculty of Law (spring 2010) is that 
all students who apply to the introductory course and meet basic requirements for 
higher education are accepted. Then, depending on their study results throughout 
the course, students can continue and enter the full law programme. Lund University 
Libraries Head Office got the Directory of Open Access Journals (DOAJ) SPARC Europe 
Award for Outstanding Achievements in Scholarly Communications, in 2009 29 . Lund 
University library has a considerable amount of e-resources with self instructions, 
open content, learning objects and Open Educational Resources (OER) (Ossiannilsson 
& Sponberg, 2010; Wheeler, 2010). During 2010, a national initiative on OER, where 
Lund University participates, is ongoing in the Swedish Royal Libraries regime, 30 
aiming to disseminate information and valorise the use of OER. The Department of 
Communication has initiated seminars, workshops and electronic handbooks about the 
use of new technology and social media; a special project on a renewed IT strategy and 
a new web are among the prioritised areas according to the Lund University strategy 
plan of 2007-2011. An initiative on Personal Learning/Web Environment (PLE/PWE) 
called live@lund.se will be implemented in autumn 2010 at the School of Economics, 
and other faculties will follow. 

As mentioned above, teacher training (CHETT) in Sweden is built on scholarships 
for teaching, learning and constructive alignment. One of the courses in this 
programme at Lund University, Learning and Teaching in Higher Education (LATHE), 


27 HEA. The Centre for Legal Education Supporting law teaching: training and teaching. Presentation at UKCLE seminar on 
teaching and learning for legal skills trainers, 16 February 2005, http://www.ukcle.ac.uk/resources/biall/clinch.html, 
availablel00730 

28 G. Salmon, Beyond distance research alliances, University of Leicester, availablel01023 
http://www2.le.ac.uk/departments/beyond-distance-research-alliance/About%20Us/staff/gilly, available 10 102 3 

29 SPARC Europe Award, http://www.sparceurope.org/news/the-directory-of-open-access-journals-receives- sparc-europe-award- 
for-outstanding- achievements-in-scholarly-communications-2009, availablel00730 

30 The national OER project on open resources, http://www.ithu. se/?page_id=9, availablel00730 


is available on the web. This will hopefully have a positive outcome, as teachers’ 
e-maturity is a prerequisite for the development of eLearning. 

eLearning has to be embedded in all levels of education, and although it probably 
sometimes is, it is not emphasised enough, and remains less integrated into the 
holistic approach of higher education. Still, eLearning at Lund University is built on 
individual drives, rather than on mainstream teaching. However, for all activities at 
Lund University, the use of new technology and social media are strongly promoted. 
Lund University has already its own YouTube channel 31 and ITunes channel. 32 
Furthermore, there is a need for reward systems and dedicated teachers in order 
to promote innovation in teaching, learning, and the use of new social media and 
collaborative approaches. At Lund University, already three times in a row in a two year 
period, educational awards for excellence in education have been dedicated to teachers 
working with eLearning in their courses. In 2009, teachers working and responsible for 
LUMA-GIS (Pilesjo, 2010) got the EADTU award of the E-xcellence Associates label, 
among other things for their innovative eLearning approach, built on personalisation, 
interactiveness, flexibility and accessibility. The other two were for innovative teaching 
and learning approaches with new media in physiotherapy in 2009, and in Chinese 
language in 2010. 

Several aspects, criteria and dimensions that are considered and implemented at 
Lund University will be discussed below. 

5.4. New eLearning specific aspects and criteria 

Several international authorities, United Nations (2010), 33 UNESCO (2010), 34 the 
Bologna process and Bologna beyond 2010 35 and Bologna 2020 36 , emphasise that 
eLearning will play an important role in reaching some of the global goals of the 
21st century, not only from the perspective of lifelong learning, internationalisation, 
globalisation and e-governance, but of sustainable development in several sectors. They 
all convey the needs of individuals in the 21st century, and the coming policies which 
lead to advantages for learners, institutions, innovations (3rd mission) and societies. 
Aceto, Dondi & Nascimbeni (2010) share their visions about learning in Europe in 
2025, emphasising the challenges of higher education as commitment to lifelong 
learning and the implementation of student-centred learning. Increased attention 
should be paid to virtual mobility in order to build intercultural dialogue, and to 
support the internationalisation of the curricula. Quality assurance guidelines in higher 
education (cf. ENQA, 2007, 2009) will play an important role for further development 
in the future. At the same time, considerations have to be made for the existing 
diversity of approaches to eLearning standards in Europe. 

Critical success factors for students are beyond the course itself, which, naturally, 
has to be on the highest level of excellence. For students, logistic excellence throughout 
the studies is fundamental. Students in eLearning courses require logistics during 
the course itself with adequate feedback, dialogue and interaction with tutors, peers 
and international top lecturers. Top-level e-resources and accessible library resources 


31 LU YouTube channel, http://www.youtube.com/lunduniversity, available 101023 

32 About LU and ITunesU, http://nywebb.blogg.lu.se/overgripande-projektstatus/, available 101023 

33 United Nation, The Millenium Goals, http://www.un.org/millenniumgoals/, available 101023 

34 UNESCO, Education for all, http://www.unesco.org/en/efa/, available 101023 

35 Bolognabeyond 2010, http://www.ond.vlaanderen.be/hogeronderwijs/bologna/actionlines/Beyond2010.htm 

36 Bologna2020, http://www.ond.vlaanderen.be/hogeronderwijs/bologna/conference/documents/leuven_louvain-la-neuve. 
communiqu%C3%A9_april_2009.pdf, available 101023 


with flexibility in time, space and languages are also vital, as is to a high extent and as 
expected, academic professionalism (Jaldemark, 2010; Ostlund, 2008). 

Personalisation is the leading concept in eLearning in the 21st century. Many 
students study at several universities around the world at the same time, and in 
addition, they have their private social communities. For those reasons, it is important 
for students to remain members of a virtual environment, for example 

Personal Learning Environment (PLE), where all logins can be visible at the same 
time. Those core critical success criteria are almost met through the E-xcellence 
Associates label, 37 with its success factors flexibility, interactiveness, personalisation and 
accessibility. The movement on OER and its use will change educational structures 
in many ways (Atkins et.al., 2007; Hylen, 2007; Wheeler, 2010). When the students 
are their own PLE as Wheeler (2010) discusses, this will dramatically change higher 
education in the future. However, the future is already here. 

Participation, production and personalisation, or the three P:s pedagogy for 
the networked society by Me Loghlin and Lee (2008), will surely be the new path 
for education in the 21st century. The three P:s are crucial within the concept of 
collaborative learning and networking. Students and teachers, or rather, mentors 
work together, participate in, and produce collaborative processes. Jaldemark (2010) 
emphasises personalisation, taking the concept a bit further. He stresses that the 
students’ learning environment is about her/his entire life-situation, comprising his/her 
living, study and learning context, as well as his/her social, economic, demographic, 
and cultural context. Jaldemark discusses it as boundless education and learning, and 
that learning environments have to meet those demands. In this context, accessibility, 
personalisation, interaction and flexibility demand an even wider and deeper 
interpretation. 

New aspects on eLearning for universities in the 21st century will surely include 
embedded eLearning in overall strategies; action plans and processes, and integration 
in ordinary quality assurance processes as in the aims of the ELQ (NAHE, 2008). 
Bonk’s (2009) paradigm ”We all learn” 38 gives additional new perspectives on 
eLearning. Furthermore, increased interest and research on learning, teaching and 
education cultures, attractive and flexible learning and teaching environments both 
on campus and virtually, blended mode, educational development, award systems for 
teachers, and recognition will be challenges for the future. 

Information, interaction and communication issues are within the first generation 
of eLearning; beyond this are the three P-paradigm discussed above; personalisation, 
participation and production (McLoughlin & Lee, 2008), added with portable learning 
according to Bonk (2009). In a European perspective, learning through technology 
and learning to work with technology in a knowledge-based society in the 21st century 
demand a diversity of approaches to eLearning standards, e-skills and e-working skills 
should be more formally appreciated and recognised as learning outcomes in courses 
and assignments, as these are skills for the future society. The concept of flexibility 
needs to be extended, not only in terms of geographical location and learning modes, 
but also in terms of language and accessibility (Bonk, 2009; Salmon, 2004, 2005). 
Open Educational Resourses (OER) (Hylen, 2007; Ossiannilsson & Sponberg, 2010; 
Wheeler, 2010), open content (Johnson et al, 2010), micro-training and informal 

37 EADTU E-xcellence Associates label, http://www.eadtu.nl/e-xcellencelabel/, available 100730 

38 web-searching in the world of e-books, eLearning and blended learning, availability of open source and free software, leveraged 
resources and open courseware, learning object repositories and portals, learner participation in open information communities, 
electronic collaboration, alternate reality learning, real-time mobility and portability and finally networks of personalised 
learning. 


or formal learning are powerful concepts in the paradigm of lifelong learning and 
innovation. They allow the individual to control the learning process and her/his 
learning environment (Ossiannilsson, 2010 & Sponberg, 2010). In a global world, both 
localisation and globalisation have to be present at the same time. In fact, we can talk 
about glocalisation’. Boundless higher education needs to be extended towards this 
direction. In the 90 's, Moore (1997) describes quality in online courses in respect of 
structure, dialogue and autonomy, alance Between structure and dialogue is crucial 
and determines how large and rich the transaction can become. The interaction in 
online based learning environments can focus either on social dimensions (Wegerif, 
1998), subject or task related dimensions (Du, Havard, & Li, 2005) or dimensions 
with combinations. Already in 1989, Moore (1989) stated that learning interaction is 
threefold: the learner and the content (learner-content), the learner and the instructor 
or mentor (learner-instructor) and the learner and other learners (learner-learner). 

Due to its aims and interactive level, the delivery and assessment of a course demands 
different kinds of enhanced technology, e-resources and e-skills according to the 
discourse and debate in the area. 

Different courseware may be used depending on the type and level of interaction: 

• Primary courseware provides documents, films, links etc. for a thorough 
presentation of a subject. 

• Secondary courseware provides environments and tools to perform learning 
activities such as forums, portfolios, and questionnaires. 

• Tertiary courseware includes the material produced by learners themselves, e.g. 
forum contributions, portfolios, project reports and documents etc. 

The development of independent learners in the 21 st century requires going 
beyond capacities; new abilities are demanded as e-investigators, e-writers and 
e-communicators and collaborators (Bonk, 2009; MacDonald, 2008; Salmon, 2004, 
2005). Flexibility is more than time and space when it comes to learning in a digital 
world. Qualifications and admission, framing, implementation and resources (Collis & 
Moonen, 2002) are important concepts. Constructive alignment (Biggs, 2003), aligning 
learning outcomes, learning and teaching activities, assessments and scholarships in 
teaching and learning are other crucial concepts in eLearning. 

5.5. How to integrate eLearning criteria in the national evaluation 
programme 

Although several initiatives in Europe have recently been conducted on benchmarking 
and quality in eLearning, there is a lack of research in this area (Ossiannilsson, 2010a, 
b). However, conclusions can be drawn from the integration of critical eLearning 
success factors in ordinary quality assurance in national and international institutions. 
One example is the ELQ-model (NAHE, 2008), which can and maybe used. NAHE 
emphasises the importance of a holistic approach and that eLearning needs to be 
integrated in overall quality assurance processes. Ten criteria are formulated and all 
ten need to be taken into consideration in a holistic perspective (NAHE, 2008). The ten 
criteria are: 

• material and content 

• structure and virtual environment 

• communication 


• cooperation and interactivity 

• student assessment, flexibility and adaptability 

• support (student and staff) 

• staff qualifications and experience 

• vision and institutional leadership 

• resource allocation 

• holistic and process aspect. 

These aspects correspond with both E-xcellence+ and ELBE. According to NAHE, 
methods that are normally used on quality assurance need to be updated and adapted 
for different forms of eLearning. Further quality aspects on eLearning need to be 
embedded in quality assurance systems. Even internal competence and information 
provision on eLearning need to be assured. Additionally, NAHE stresses that internal 
working methods need to be adapted according to special conditions that boundless 
education implies and demands (2008, p. 8). 

Another strong approach would naturally be the integration of eLearning in the 
ENQA guidelines for quality assurance. 39 Research on educational cultures both 
virtual and blended mode approaches are under development and of large interest 
around the world. Experiences and lessons learned from European and international 
eLearning benchmarking initiatives should be considered as ‘good examples’ for 
quality enhancement with regard to educational management, content and services 
at universities (Bacsich, 2009c; Comba & Ossiannilsson et.al., 2010; Ossiannilsson, 
2010a, b; Ossiannilsson & Landgren, 2010a, b, c; Ubachs, 2009; van Vught, 2008a, b; 
Williams & Rotheram, 2010;). Trucano (2010) argues that development and success 
in the use of new technology and eLearning demand support and responsibilities by 
national agencies (compare the former role of the Swedish Net University and NSHU). 
The ReViCa initiative and its content, dissemination and valorisation (Schreurs, 2009) 
needs to be taken into consideration as well as ongoing discourse and debate in the area 
by such scholars as Andersen, 40 Bates, 41 Bonk, 42 Flate Paulsson, 43 Salmon, 44 Wheeler, 45 
just to mention some. Furthermore, the work of European and International networks 
in the field (e.g. EADTU, EDEN, 46 EFQUEL 47 NMC 48 and ICDE 49 ) has to be considered 
and implemented in quality assurance strategies, policies and plans of higher education 
institutions. Additionally, experience and roadmaps from partners involved in the 
recent initiatives by EADTU and ESMU provide good examples on how quality 
eLearning indicators and critical success factors can be integrated in ordinary quality 
assurance (Williams & Rotheram, 2010). According to statements by the president of 
ENQA, the speakers and the participants (unpublished communication) during the 
workshop in Sigtuna in 2009 (ENQA, 2009), it is, and has to be established that the 
“e” is here to stay for learning and teaching in the 21st century, and the challenges for 
higher education just has to be taken on board. 

39 ENQA (the European Association for Quality Assurance in Higher Education) disseminates information, experiences and good 
practices in the field of quality assurance (QA) in higher education to European QA agencies, public authorities and higher 
education institutions. 

40 T. Andersens homepage, Athabasca University, Us, http://cde.athabascau.ca/faculty/terrya.php, availablel01023 

41 E-learning and distance education resources. Blog by T. Bates, http://www.tonybates.ca/, availablel01023 

42 C. Bonks eLearnig world, Professor, http://php.indiana.edu/~cjbonk/, availablel01023 

43 M. Flate Paulsson, NKI, Professor of online education, and President of EDEN, http://home.nki.no/morten/ 

44 G. Salmon, Beyond distance research alliances, University of Leicester, availablel01023 http://www2.le.ac.uk/departments/ 
beyond-distance-research-alliance/About%20Us/staff/gilly, availablel01023 

45 S. Wheeler, the complete works, http://www2.plymouth.ac.uk/distancelearning/steve.html, availablel01023 

46 EDEN, http://www.eden-online.org/eden.php, availablel01023 

47 EFQUEL , http://www.qualityfoundation.org/, availablel01023 

48 New Media Consortium, http://www.nmc.org/, availablel01023 

49 International Consortium of Distance Education, http://www.icde.org/ 


Results from the benchmarking initiatives conducted by Lund University showed 
that advantages and benefits with eLearning and e-courses can and probably also have 
to be transmitted to Campus courses and to be integrated in all areas, as a natural part 
of education in the 21st century. The goals and working methods of quality assurance 
processes involve a sustainable development, internationalisation, boundless education, 
constructive alignment and scholarships for teaching and learning, critical colleagues 
and peer reviews, and continuous benchmarking at national and international 
levels. The discourse and debate in the field explicitly shows the need for innovation 
within teaching and learning in higher education and reconstruction of universities 
concerning how courses are designed and offered (Bates, a, b, c; Batson, 2010; Nygren & 
Larsson, 2010; Ossiannilsson & Landgren, 2010a, b, c; Robinson, 2010). 

5.6. Conclusion 

Through the findings from participating in benchmarking processes and the ongoing 
discourse and debate in the area, it seems that higher education will need to meet 
challenges at different levels. First, to meet the new multitasking generation of 
learners in the 21st century with new skills, especially e-and mobile skills. Second, 
living in a global world demands boundless and mobile education within the lifelong 
learning context and sustainable development perspectives. Third, universities need 
to function within new innovative structures to meet collaborative learning processes. 
Education, teaching and learning methods need to be reconstructed, and teachers’ 
e-maturity developed. Research also shows strong connections between successfully 
implemented eLearning and strong conscious management and leadership at all levels 
(Ossiannilsson, 2010a, Ossiannilsson & Landgren a, b, c). 

Taking part in benchmarking has proved valuable in several ways, as expressed 
both by ESMU (van Vaught, 2008a, b) and Moritary (2008). It has had implications 
for internal changes. Through active participation in benchmarking exercises, Lund 
University has established a reputation for sharing experience both at the internal and 
the international level. The final conclusion is the value of continuous benchmarking 
exercises in different areas for the purpose of quality assurance and enhancement 
in higher education, and, in this respect, of meeting the demands of individuals for 
boundless education. 

The Sigtuna workshop concluded and agreed that the accreditation, audit and 
assurance processes of e-learning should be integrated in national framework and not 
be evaluated separately. This was also suggested by NAHE (2008). There is a need, 
however, for methodological development within quality assurance agencies. At the 
same time, there are demands for increased cooperation between national, European 
agencies and international agencies, as eLearning enhances the development of 
boundless education. Hopbach (ENQA, 2009) concluded the Sigtuna workshop 2009 
with three statements: 

1. It is important to meet and discuss quality assurance at the European level and 
between different stakeholders in the educational sector 

2. There is a need for a “common” definition of eLearning, to so to say know what we are 
speaking about, and have a common language. 

3. And finally, eLearning must be an integrated part in higher education and quality 
assurance has to cover all aspects of eLearning. Lurthermore, there are demands of 
expertise by the evaluators. 


Additionally, it is important to mention that the challenges of eLearning have to be 
embedded beyond and boundless, but with an innovative and creative approach. 
However, the most important challenges are to consider the conditions that will 
facilitate optimal and powerful learning processes for students, including the entire 
process from interest in university studies to application, studies, exams and credits, 
and all the way to alumni. Professionalism in all aspects of learning, eLearning and 
mobile learning is a key condition for the achievement of boundless and academic 
global learning environments in the 21st century. NAHE (2010b) has recently 
conducted a survey on distance education in Sweden, probably bringing back results 
relation to the ELQ model for the consideration of national agencies. 
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Conclusion 


As the integration of information and communication technologies (ICT) in modern 
educational activities is creating alternative study paths, eLearning has become one of 
the most prominent concepts within the higher education institutions of today. 

eLearning has become a particularly attractive educational method, as the 
use of web-based tools reduces the costs of sharing vast amounts of data, reduces 
communication barriers and geographical distance gaps between individuals, increases 
academic mobility in higher education, provides people with disabilities to have 
better access to higher education, and allows smaller institutions to gain international 
visibility through study programmes online. 

eLearning faces, however, certain challenges. As eLearning is dependent on the ICT 
as the primary teaching and learning tool, the prerequisite for an effective eLearning 
process is that learners have sufficient ICT skills, and programme providers have 
sufficient professional competence and adequate educational strategies to manage 
distance programmes with the help web-based tools. Thus, it should be emphasised that 
eLearning is efficient only as long as it the learning content is updated regularly and the 
teaching methodology used in the distance programmes helps the learners to master 
the learning material and to acquire knowledge. In short, eLearning must respond to its 
stakeholders’ needs. This involves quality assurance agencies. 

The articles in this publication show that there is only very little experience in the 
assessment of eLearning in Europe. In fact, eLearning quality is rarely included as 
a regular or integral part of national quality reviews, nor is any emphasis placed on 
the ESG. In short, quality assurance of eLearning remains yet to be developed. Thus, 
quality assurance agencies should adapt to the alternative learning and teaching 
methods and to the associated challenges that the ICT create, and develop assessment 
standards and benchmarks that would help the traditional universities make the 
necessary internal transformations and enable methodical evaluation and improvement 
of eLearning. 

It is therefore vital to establish a solid quality assurance system in Europe for greater 
accessibility to and quality of eLearning. eLearning should not be evaluated separately, 
but as an integrated part in higher education. Moreover, quality assurance has to cover 
all aspects of eLearning. There is a need for a common definition and understanding 
of the concept of eLearning, a need for a “common language” that would help higher 
education institutions and quality assurance agencies strive for the same goal. To meet 
this goal, it is important to meet and discuss quality assurance at the European level 
and between different stakeholders in the educational sector and to provide adequate 
training for academic professionals, higher education providers and quality evaluation 
experts. 


ANNEX 1 

- Programme of the workshop 


ENQA Workshop 

QUALITY ASSURANCE OF E-LEARNING 
7-8 October, 2009 

Hosted by the Swedish National Agency for Higher Education (NAHE) 

Venue: 

Kristina Konferens & Hotell 
Rektor Cullbergs vag 1 
193 23 Sigtuna 
Sweden 

WORKSHOP PROGRAMME 

DAY 1 
Wednesday 
7 October, 2009 

12:00 Lunch 

13:00 Welcome addresses 

Lena Adamson, Secretary General, NAHE 
Achim Hopbach, President, ENQA 

Introduction to the workshop and the ELQ-report - aspects and criteria, 

Per Westman, NAHE 

13:45 The UNIQUe (EFQUEL) approach to quality in E-learning, 

Annemie Boonen, European Foundation for Quality in E-learning (EFQUEL) 

The E-xcellence (EADTU) approach to quality in E-learning, 

George Ubachs, European Association of Distance Teaching Universities 
(EADTU) 

15:15 Coffee break 

15:45 Discussion theme 1 - Quality assurance specific for E-learning 

What are the new aspects and criteria specific for E-learning that need to be 
considered? 


Panel discussion with higher education institution and open university 


representatives. 


Panelists: 

Goran Karlsson, KTH Royal Institute of Technology, Sweden 
Denise Kirkpatrick, The Open University, UK 
Josep Llados, The Open University of Catalonia, Spain 
Ebba Ossiannilsson, Lund University, Sweden 

Chaired by Per Westman 

-17:15 

17:30 Pick-up from the hotel for a guided walk in the medieval town of Sigtuna 
19:30 Dinner at the workshop venue 


DAY 2 
Thursday 
8 October, 2009 

9:00 The UK approaches to quality in e-learning, as seen from the 
HE Academy/ JISC benchmarking programmes 

Paul Bacsich, Matic Media Ltd, UK 

9:45 ELQ report, Policy issues for quality assurance agencies 

Per Westman, NAHE, Sweden 

10:00 Coffee break 

10:30 E-learning in the context of the Standards and Guidelines for the Quality 
Assurance in the European Higher Education Area (ESG), 

Josep Grifoll, ENQA Board member, AQU Catalonia, Spain 

11:00 Discussion theme 2 -Challenges for quality assurance organisations 

How can the e-learning criteria be implemented in national evaluation 
programmes? 

In the “ELQ” report, the following issues are emphasised when integrating the 
assessment of E-learning into the general assessment framework: 

• The integration of the quality assessment of E-learning to national 
reviews requires specific competence within the assessing 
organisation. 

• Cross-boundary education requires cooperation and exchange of 
knowledge between quality assurance agencies in order to harmonise 
and safeguard strategies and policies for the quality assurance of 
E-learning. 


• Extensive methodological development is necessary in order to adapt 

common methods for the assessment of the quality of E-learning in 
higher education. 

Panel discussion with representatives from quality assurance agencies. 
Panelists: 

Esther Huertas Hidalgo, AQU Catalonia, Spain 
Fred Mulder, NVAO, the Netherlands 
Yuri Rubin, AQA, Russia 

Chaired by Josep Grifoll 

12:15 Lunch 

13:15 Discussion Theme 3 - Future cooperation 

How can European level cooperation and continuous knowledge exchange be 
built between quality assurance agencies, organisations with experience in the 
quality of E-learning and with other stakeholders? 

Panelists: 

Love Hansson, European Students’ Union (ESU) 

Carl Holmberg, International Council for Open and Distance Education 
(ICDE) 

George Ubachs, EADTU 
Chaired by Achim Hopbach 
14:30 Conclusions 
14:45 End of Workshop and coffee 



ENQA held a workshop in coordination with the National Agency for Higher 
Education, in Sigtuna, Sweden in October, 2009. The workshop created a dialogue 
between institutions, quality assurance agencies, students and other stakeholders 
who are directly affected by the quality of E-learning. This report gives a general 
overview of the matters discussed and challenges faced within the sector of quality 
assurance in E-learning. 



